SUBSTITUTE APPLICATION 



COORDINATED AUDIO AND VISUAL OMNIDIRECTIONAL RECORDING 



FIELD OE THE INVENTION 

This invention relates to simultaneous recording of panoramic visual 
data and panoramic audio data and the application of the directed 
audio (or radiated) signal to focus the visual recording on the source 
of the directed audio signal. 

BACKGROUND AND INFORMATION DISCLOSURE 

This invention relates to simultaneous recording of panoramic visual 
data and panoramic audio data and the application of the directed 
audio (or radiated) signal to focus the visual recording on the source 
of the directed audio signal. 

BACKGROUND AND INFORMATION DISCLOSURE: 
The use of video photography has evolved steadily over the past 
forty years in its application to surveillance situations and 
environments. Applications include stationary systems such as are 
found in commercial establishments, and on moving bases such as 
robots, m submarines, vehicles, etc. 



For example, U.S. Patent 6,016,385 to Yee et al discloses a real time 
remotely controlled robot having a head with a pair of television 
cameras (eyes) and a pair of microphones (ears) and a pair of arms 
mounted on a mobile trunk. An operator located at a command 
center receives audio and visual signals from the robot and, in 
response, issues commands that control movement of the trunk and 
arms to perform tasks. 

U. S. Patent 4,453,085 to Pryor discloses an electro-optical system for 
monitoring the positioning of a movable arm. 

U. S. Patent 4,604,559 to Hawkes et al discloses a robot controlled by 
a user wherein an audio signal is fed back from the robot to the user 
regarding force and movement parameters of the robot operation. 

U. S. Patent 5,299,288 to Glassman et al discloses an image directed 
robot system for surgical applications including an optical tracking 
camera supplying data to the controlling processor. 

One of the major developments in video photography in recent years 
has been the panoramic camera. According to this technology a lens 
with a 360O field of view stores the entire field in memory and 
various areas are selected and projected according to the interests of 
the viewer. This technique has many applications, for example, in 



real estate advertisements where features of the location are 
selected and enlarged for detailed examination by the viewer. 

U.S. Patent 5,920,337 to Glassman et al discloses an omnidirectional 
visual image detector and processor (incorporation herein by 
reference). There is described a reflective round lens (either a cone 
or spherical section) which reflects the object beam from a 
surroimding field of view (encircling the lens) to a direction parallel 
to the axis of the rotund lens. The beam then passes through an 
objective lens and is focused onto a CCD (charge coupled device) 
where image data is detected and processed for storage or image 
presentation on a monitor. 

In the context of this specification, the term, "visually examine" 
means that, a narrow field of view is selected covering a few degrees 
from an entire 360° field of view recorded by a panoramic video 
camera. 

SUMMARY OF THE INVENTION 

It is an object of this invention to develop a surveillance device that 
enables a user to visually examine a location that has a source of a 
signal. The signal is an audial signal in one version or an 
electromagnetic signal (radio, microwave, light, infrared, etc) in 
another version. 



This invention is directed toward an omnidirectional camera having a 
lens for viewing a 360° field of view; a visual memory for storing 
visual data at addresses representing the field of view; an 
omnidirectional detector for detecting the direction of location of 
signal source. A display selection circuit is provided for selecting 
visual data from an address in the visual memory corresponding to 
the direction of location. The visual data is displayed on a system 
monitor. 

In one version, a directional audio detector and a panoramic OD 
(charge coupled device) camera installed in the center of a site, (e.g., 
store, prison yard, battlefield, etc.) The camera is continually 
generating visual data of a 360° view of the site. If a soimd is 
generated from a location in the site, (an intruder, a shout, a shot, 
etc.) Then the audio receiver of this invention will detect the sound 
and its direction relative to the camera and direct the camera to 
display that portion of the stored data representing the visual image 
of the source. 

Another object of the invention is to provide a panoramic audio 
detector that is capable of receiving an audio signal fi*om a source 
located any where on a 36CP circimiference around the audio 
detector, "point" to the location of the source of soimd and then 
cimplify the audio signal for more detailed examination of the sound. 



The data corresponding to the direction of the source of sound is 
used to control the display selected from the visual data stored by 
the camera. 

In one embodiment, the panoramic audio detector of this invention 
comprises a plurality of audio detectors, strung one after the other 
on a circle. Each audio detector comprises a piezo electric element, a 
flip-flop, and coupling circuitry between the piezo electric element 
and its respective flip-flop. The coupling circuitry enables the piezo 
element to set its respective flip-flop at the instant a sound impulse 
is received by the piezo-electric. The direction of the source of sound 
will determine which flip-flop is set before all the other flip-flops. 
The first flip-flop to be set will store its address (referred to herein 
as the "source address") in a central register and the central register 
will ignore signals from all other flip-flops during the sampling 
period. 

The register will thereby have stored an address corresponding to 
the direction of the source. This address is used to select camera 
data from the panoramic camera for localized display of the source of 
sound. 

The phase of the soimd signal reaching the detector closest to the 
source will be ahead of the phases of the same sound signal reaching 
the other (more distal) detectors. The invention includes an audio 



"focusing" circuit that adjusts the phase of the signal from each 
detector so that each signal from the respective detector is in phase 
with the detector nearest the source of sound. An adder adds 
together all of the phase adjusted signals to generate a "summation 
sound" signal from the source. 

In another embodiment, a focusing circuit coupled to the panoramic 
camera displays the image of a location in response to motion in that 
location. The panoramic viewing system has two video memories, 
and a comparator that is configured to compare point by point image 
data at each memory address of one visual memory to data at the 
corresponding addresses of the other visual memory. A difference 
between a signal stored at one address in one memory and the signal 
at the same address in the other memory indicates motion at the 
location and this difference causes the circuit to display the image of 
the location where motion has occurred. . 

BRIEF DESCRIPTION OF THE FIGURES: 

Fig. 1 shows an audial omnidirectional detector of this invention. 

Fig. 2A and 2B show one embodiment of a selection means. 

Fig. 3 is a timing diagram of the embodiment of fig. 4. 

Fig. 4 shows another embodiment of a selection technique. 

Fig, 5 Show an audial array having two detectors. 

Fig. 6 shows a version of the invention being a panoramic 

teleconference table. 



DESCRIPTION OF ILLUSTRATIVE EXAMPLES: 



Turning to a discussion of the drawings, fig. 1 is a schematic diagram 
of one version of the invention 10 in which the directional function 
of the panoramic camera responds to a sound coming from a source 
at a location on the circumferential field of view. 

There is shown an omnidirectional visual imaging system 13 
including an array of "sub " lenses 12, focusing lens 14 and charge 
coupled device array 16. The omnidirectional visual imaging system 
13 is of the type disclosed by U.S. Patent 5,920,337 to Glassman 
Which is incorporated herein by reference. 

An audio detector system 1 1 is shown as a string of detectors 30 
moimted on a circle that is coaxial with an axis 18 of the lens 12. 
The detectors 39 are preferably piezo electric, capacitive, 
semiconductor elements or similar pressure sensitive devices well 
known in the art. 

Each detector 30 is coupled to a selection circuit 49 that responds to 
the first detector to receive a soimd wave radiated from a source 34 
by shutting off the signals from all of the other detectors. 
Soimd generated by a source 34 from a location on the 
circumferential field of view reaches first the detector 30A closest to 
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the source 34 before it reaches the other detectors thereby shutting 
off the signals from ail of the other detectors. An address register 36 
registers the address of the closest detector 3 OA and ignores the 
signals from the remaining audial detectors 32. The registered 
address is applied to display control circuit 38. The visual display 
circuit 38 selects the visual data from the corresponding address in 
the video memory 40 and displays an image of the location on the 
system monitor 42. 

Also displayed is a niunber indicating the azimuth direction of the 
soiurce of sound relative to the camera. 

One embodiment of a selection circuit 49 is shown in fig. 2 A showing 
an array (ring) of audial detectors 30. Fig. 2B shows a portion of 
circuit 49 to better advantage than fig. 2A. The output terminal of 
each audial detector 30 is connected to an input terminal of an AND 
gate 51. Each detector 30 has its own AND gate 51. 

The output terminals of all AND gates 51 have two cormections: 

(i) One connection 63 is to an address register (not shown) which 
activates a portion of the CXD to send an image signal representing 
that portion of the field of view to a video monitor. 

(ii) The second coimection of all output terminals 53 of all the AND 
gates 51 is connected to a bus 57. Bus 57 is coimected to an input 



terminal of a single inverter 55 whose output terminal is connected 
to bus 59. Bus 59 is connected to a second input terminal 61 of all 
AND gates 51. 

Initially there is no soimd incident on any of the detectors so that the 
input signal to all of the gates 51 is zero. When a soimd signal is 
radiated from a soimd source, the closest detector is the first to 
receive the signal and sends a positive signal to one input of its gate 
Since the signal on the second input terminal is zero (not negative) 
the first gate to receive the signal from its detector 30 will send a 
signal to the address register 63. The positive signal from the first 
gate to receive the soimd signal is inverted by inverter 55 and the 
negative signal is appUed through the second bus 59 to the second 
input terminal of all the remaining gates thereby turning off all 
remaining gates. 

The video memory 40 reads the selected address from register 63 
and applies the image data on monitor 42. 

An alternate scheme for detecting the direction of the sound is 
shown in fig.4. There is shown a ring of audial detectors 30 arranged 
relative to the omnidirectional lens. The lens is not shown in fig.4 
but is arranged coaxial with the audial detector array as discussed 
with regard to fig. 1. 



Each detector 30 of the array of detectors forms a pair of detectors 
with one member of each pair being positioned on a circumferential 
boundary oppositely from another member of the pair. For example, 
detectors 30A and SOB form one pair, detectors 30C and 30D form 
another pair. A plurality of difference circuits 37 is provided 
although only two difference circuits are shown in fig. 4. a difference 
circuit for each pair of detectors, one pair of detectors for each 
difference circuit 37. 

Fig. 4 shows inverters 39 (two are shown), 

one Inverter for each pair of detectors, one pair of detectors for 
each inverter; 

a rectifiers 49, one rectifier 49 for each pair of detectors 30, one 
pair of detectors 30 for each rectifier 49; 

An address register 41 having a plurality of address input terminals, 
one address terminal for one difference circuit and one difference 
circuit for one address terminal; 

one member of each pair of detectors 30 is connected to transmit an 
image signal to an input terminal of one of the inverters 39; 



an output terminal of the inverter is connected to an input terminal 
of one of the difference circuits; 

another one of each pair of detectors is connected to another input 
terminal of its difference circuit; 

each one of the rectifiers 49 has an input terminal connected to an 
output terminal of the difference circuit 37; 

each one of the rectifiers 49 has an output terminal connected to one 
of the input terminals of the address register 41. 

the address register has an output terminal configured to emit an 
address signal responsive to a null signal from one of the rectifiers. 

The timing diagrams of figs. 3.1-- 3.7 show the signals generated by 
a soimd pulse 46 incident on the array of detectors 30. Detector 30C 
is the "leading" detector where the phase shift of the incoming wave 
will be assumed to be zero as shown in fig. 3,4. Figs. 3.1 and 3.2 
show that the signals received by detectors 3 OA and 30B have 
shifted an equal amount, 0, so that the difference between their 
signals (fig. 3.3) is zero. The inverted signal, Fig. 3.5, from detector 
30D has shifted by 20 thereby giving pulse 3.6. 



Fig. 3.7 is a plot of the peak output of each difference circuit 37 vs. 0 
Interpolation of the plot to Zero yields the value of 0 being the 
direction of the sound 46. 

When no sound is incoming, the output S3 = 0 (fig. 3.7). When a 
sound wave 46 is incident on the array 30, then a finite pulse 3.7 is 
generated from the respective one of the difference circuits that 
stores the signal in the register at its azimuthal address 0 Selection 
circuit 45 then applies the address 0 to video memory 46 to locate 
the video data corresponding to the source of the sound 46. Video 
memory 46 is connected to CCD array 16 for storing an image signal 
generated by the CXD 36 responsive to the visual image of the 
surrounding field formed by omnidirectional lens 13. The selected 
image data is presented on video monitor 42. Not shown in fig. 2 but 
shown in fig. 1, the axis of the audial array is coincident with the axis 
of the omnidirectional lens 13. 

Fig. 5 shows another embodiment of the invention being a system 
which relies on the phase difference of the sotmd wave between a 
pair of audial detectors 50 and 52, to present on a video monitor 42 
an image of an area of a field of view formed by an omnidirectional 
lens 13 on a CCD array and stored in video memory 40. 

If P represents the period between the arrival of the earliest 
detectable portion of a soimd signal, (such as a shot) to each detector. 



The velocity of sound is C Then the angle, 0, of the direction of the 
sound relative to the audio detector line is 

cos 0 = PC/13 

In practice, visual data is stored in the visual memory 40 at 
addresses corresponding to the angle 0. The phase detector 51 
measures the time period P being the difference in times of arrival of 
the signal from source S to first audial detector 50 and detector 352. 
The value of 0, calculated by the formula gives the address of the 
image data for generating a local image to be displayed directly on 
the monitor 42. 

Several embodiments have been described of a surveillance system 
being a combination of audial detectors that sense the direction of an 
incoming sound wave and a visual system that presents an image of 
the area includes the source of the soimd. 

Other variations and modification of this invention may be 
contemplated after reading the specification and studying the 
drawings that are within the scope of the invention. 

For example other forms of lens (other than an omnidirectional lens 
may be used to collect the image data. 

Fig. 6 shows an embodiment for application in televising a 
conference. The conferees 62 (seven are shown) are located (seated) 



at locations around a panoramic camera 64 positioned on a support 
65 (e.g,, table). Each location has a microphone 66. Each location is 
viewed through a corresponding section of the lens of camera 64. 
When the voice of the conferee 62 nearest a particular microphone 
64 at the conferee's location is detected by the microphone 64, the 
voice signal is amplified and transmitted to signal selection controller 
70 which conditions the panoramic camera 64 to project the image 
formed by the corresponding section 68 on the monitor 72. The 
signal selection controller 70 is shown in phantom in fig. 6 because it 
is preferably located under the table 65. 

In another arrangement, the signal selection means includes an array 
of buttons (switches) on a console 73. Each switch, when closed, 
connects a selected one of the microphones 62 to a source of power 
enabling the selected microphone 62 to convert sound to an electrical 
signal for transmission to the monitor coincident with selection of the 
corresponding section of the field of view for viewing on the monitor. 
In this arrangement, selection of the microscope occurs 
simultaneously with the selection of the video signal for viewing 
corresponding to the selected microphone. 

In other versions: 

the system is moimtable on a cart or robot and transportable into an 
otherwise inaccessible environment. 



the principles of the invention are adaptable to underwater 
surveillance. 

the system may be adapted to use at night by using a visual system 
that is sensitive to infrared. 

The audial detector in various embodiments include piezoelectric 
detectors, capacitive detectors and semiconductor detectors such as 
are well known in the art. 

I therefore wish to define the scope of the invention by the 
appended claims. 



